Dual pointer management method using cooperating input sources and efficient dynamic coordinate remapping

ABSTRACT

The pointer management technology establishes a protocol and method for dual pointer management in both absolute input mode and relative input mode. The method defines a set of properties/constraints for contextual dynamic remapping between input sensor coordinates and target screen coordinates. The remapping of the left pointer (respectively the right pointer) depends on the position of the right pointer (respectively the left pointer) in the target screen space. This inter-dependence enables a more flexible and more powerful interaction as it exploits the contextual layout to re-estimate the remapping transformations at each instant.

BACKGROUND

The present invention relates generally to methods, controllers andcontrol apparatuses for generating and displaying pointers or cursors ona display system in response to input from an input device. Moreparticularly, the invention relates to methods, controllers and controlapparatuses that respond to plural input sources and thereby generateplural pointers or cursors.

A variety of new wireless or corded controllers providing new andadvanced interaction methods are being introduced for consumerelectronics products. The new trend aims at breaking away from thetraditional button-based approach and at enabling a new user experiencethrough the use of multiple sensors such as touchpads, touch screens,gyroscopes, accelerometers, capacitive sensors and the like. In thetelevision arena, this trend is a response in part for new applicationmarkets linked to Internet-based services, multimedia entertainment andgaming that are being deployed.

One common and universal interaction needed for applications relates tothe ability to point and select elements displayed on a graphical userinterface of a display system. This type of interaction relies on theuse of a graphical pointer or cursor that the user can control via atouch screen, touchpad or gyroscopic input, for instance. Mostpointer-based interaction systems use one single pointer and, in effect,mimic the interaction that takes place in the computer world with amouse pointer.

Looking at its popular and widespread adoption, the single pointerapproach has certainly proven its convenience and value over the lastpast decades. However, it is argued here that many tasks could greatlybenefit from a dual pointer interaction to provide users with addedconvenience and increased speed of interaction. In dual modeinteraction, users would typically use both hands (each hand controllingone pointer). Targeted tasks for improvement include in particular:

-   -   text entry using a virtual keyboard displayed on the interface;        and    -   multiple object selection among a collection of objects        displayed on the interface.

Dual pointer interaction can be implemented simply by using two inputsources (e.g. two touchpads) and by managing the pointers independently(in either absolute or relative input mode). Independent methods includebasic pointer overlay on the whole interactive screen (i.e. two pointerson a common area) or basic split of the interaction area shown on adisplay in two static sub-areas (i.e. one area defined for eachpointer).

However, independent dual pointer management approaches have drawbacksand limitations when it comes to convenience, usability and/or precisionof designation. These limitations are linked to two fundamentalinteraction observations. First, it is confusing for users to controlpointers that can cross each other. A pointer controlled with the righthand should preferably remain on the right hand-side of the left handoperated-pointer and vice-versa. Second, it may be frustrating for usersto force them to use the left pointer for one set of actions and to usethe right hand for another set of actions (e.g. a split keyboard) in thecase where a pointer is confined to a specific area.

This invention effectively overcomes those limitations using a novelinter-dependent dual pointer management method.

SUMMARY OF THE INVENTION

The technology disclosed herein enables a more intuitive, moreconvenient and efficient interaction for dual pointer-based input bydefining a novel pointer management method that uses a cooperative modelbetween the two input sources (e.g. two touchpads, multi-touch touchpadwith two input areas, two joysticks) via an efficient contextual dynamiccoordinate remapping approach.

The approach enforces non-crossing left and right (or top and bottom)pointers while enabling users to point at any objects with either one ofthe pointers at any time, thus bringing convenience and intuitiveness tothe interaction. It also enables users to interact at greater speedsince pointing movements can be more readily optimized by the user foreach specific situation.

The disclosed embodiments target particular tasks such as text input andmultiple object selection, although other applications are possible withthe technology disclosed herein. The dual pointer concept can beextended to multi-pointer applications having more than two inputsources and more than two pointers.

As will be more fully explained herein, the disclosed technology hasnumerous advantages, including but not limited to the following:

-   -   1. Defines dual pointer management protocol that enforces        non-crossing pointers (e.g. left and right pointers),    -   2. Targets principally dual absolute pointer modes but core        principles applicable to relative pointing as well,    -   3. Defines an inter-dependent remapping process that uses left        and right pointer information to convert input coordinates (e.g.        from one touchpad) to output coordinates (e.g. to screen),    -   4. The interdependent remapping parameters are re-estimated        after each left and/or right finger activity (contextual        remapping as opposed to fixed remapping),    -   5. The remapping functions provide motion continuity and        smoothness via contextual stickiness at the input sensor edges,    -   6. The method supports smooth switching from one-hand operation        to two-hand operation and vice-versa,

In accordance with one aspect, the disclosed technology provides anapparatus for generating plural pointers on a display screen. Theapparatus comprises a multi-position input system receptive of positioninput data from a plurality of input devices, and a pointer generationsystem that generates plural pointers for display on said display screeneach at a different pointer location. The apparatus further comprises amapping integrator processor configured to compute said differentpointer locations by taking into account the position input data fromall of said plurality of input devices in computing each pointerlocation. The mapping integrator processor being further configured tocompute said different pointer locations such that each location lies ona different side of a dynamic boundary that is adjusted by the processorbased on knowledge of the input data from said plurality of inputdevices.

In accordance with another aspect, the disclosed technology provides amethod of generating plural pointers on a display screen that areresponsive to plural input devices. According to the method, a firstposition signal is received from a first input device; a second positionsignal is received from a second input device; additional positionsignals are received if more than two input devices are provided. Theposition signals are electronically processed to calculate first andsecond pointer positions within a predefined display coordinate systemassociated with a display screen. Additional pointer positions may becalculated if more than two position signals are received. Thecalculated pointer positions are different from one another and each isbased on both of at least the first and second position signals. Firstand second pointers are generated on the display screen, the firstpointer being displayed at the first pointer position and the secondpointer being displayed at the second pointer position. Additionalpointers may be generated if more than two pointer positions werecalculated.

In accordance with the method, the processing step that calculated thefirst and second pointer positions is performed such that the first andsecond pointer positions are confined to different sides of a dynamicboundary within the display coordinate system, wherein the dynamicboundary is computationally adjusted based on the first and secondposition signals.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an exemplary remote control device having two separate,clickable touchpads, useful in implementing the control technology ofthe invention;

FIG. 1B is an exemplary remote control device having a singlemulti-touch, clickable touchpad, useful in implementing the controltechnology of the invention;

FIG. 2 is an exemplary illustration of the control device of FIG. 1A,showing use in controlling plural pointers that are mapped to the samescreen area of the display screen. This figure illustrates one reasonwhy user confusion can result in dual pointer systems;

FIG. 3 is an exemplary illustration of the control device of FIG. 1A,showing use in controlling plural pointers where the left and rightpointers are confined to two fixed sub-areas of the screen; sub-areasmay or may not overlap;

FIG. 4 is an exemplary illustration of the control device of FIG. 1A,showing an example of proximity conflict in the case where the left andright pointers are confined to two fixed sub-areas of the screen;

FIGS. 5A and 5B are exemplary illustrations of the control device ofFIG. 1A, illustrating how the addressing space for the right hand sidepointer is bound on or confined to the left side by the left hand sidepointer x-axis position; and the addressing space for left hand sidepointer is bound on or confined to the right side by the right hand sidepointer x-axis position, where the boundaries move dynamically;

FIG. 6 is an exemplary illustration of the control device of FIG. 1A,showing boundary conditions for a first embodiment of the dual pointermanagement technique;

FIG. 7 is an exemplary illustration of an alternate embodiment of thedual pointer management technique where the dynamic split permits mildpointer overlay;

FIG. 8 is an exemplary illustration of yet another embodiment of thedual pointer management technique where the dynamic split enforces outerlimit restrictions to enhance effective pointer resolution;

FIG. 9 is an exemplary illustration of another embodiment of the dualpointer management technique where, in relative mode, the displacementoffset in the screen space is more important for the right pointervis-à-vis the left pointer due to larger dynamic area of the rightinteractive area;

FIG. 10 is a hardware circuit block diagram of an exemplary embodimentof a control apparatus working in conjunction with a display screen;

FIG. 11 is a flow chart diagram showing exemplary operations performedby the CPU (or processor) and graphics generator of FIG. 10 to effectthe functionality provided by the control apparatus;

FIG. 12 is a functional block diagram illustrating the exemplaryfunctionality of a control apparatus for generating plural pointers on adisplay screen;

FIG. 13 is a functional block diagram illustrating an alternateembodiment of the control apparatus that employs a multi-position inputsystem.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The technology of the innovation can be used with a variety of inputdevices enabling dual pointer or multi-pointer input. For illustrationpurposes, two exemplary input devices are shown in FIGS. 1A and 1B. Thedevice illustrated in FIG. 1A is a remote controller that features twoseparate clickable touchpads. The device illustrated in FIG. 1B is aremote controller that features a single multi-touch surface that cansimultaneously discriminate among multiple touch inputs. It will beappreciated that these are simply two possible exemplary embodiments;other embodiments using other types of control sensors are possible. Forexample, a dual pointer device may be implemented using multiplejoysticks, trackballs, proximity sensors or the like.

To better understand the concepts involved in the innovation, adiscussion of various pointer management methods will now be presented.Pointer management methods rely on the type of mapping that existsbetween the pointing device space of (e.g. touchpad surface) and thetarget area of the screen. This disclosure will focus initially onabsolute pointing methods but the basic concept introduced here can bepertinent to relative pointing as well. The innovation employs a dynamicapproach to dual pointer or multi-pointer management, in contrast toconventional approaches (overlay method, basic split method), which eachuse a static strategy.

As will be understood from the description herein, the innovativetechniques described here apply to any input device(s) providing two ormore separate addressing spaces; this can be implemented, for example,using two or more separate sensors (e.g. two touchpads of FIG. 1A) orusing a single sensor with two or more separate areas (one multi-touchtouchpad of FIG. 1B) with two logical areas.

Referring to FIG. 1A, the exemplary remote controller features twoclickable touchpads 20, 22 and a set of application-definable buttons24. Device orientations (landscape, portrait) and hand positions areautomatically detected via capacitive and accelerometer sensors.Referring to FIG. 1B, the exemplary remote controller features a single,multi-touch touchpad 26. If desired, certain designated regions on thetouchpad 26 can be assigned to a set of application-definable buttonsthat, when tapped, function in the same manner as the buttons 24 of theFIG. 1A embodiment.

Problems with Overlay Pointer Methods

As seen in FIG. 2, the overlay method uses a 1-to-1 mapping between atarget screen area 30 and any of the two touchpad areas 20, 22. The usercan point to any location in the screen area by using indifferently theright or left pointer. (In FIG. 2, and in subsequent FIGS. 2-9, theright and left pointers are graphically depicted as partial images ofthe right and left human thumbs: right pointer 32 and left pointer 34).The major drawback of the overlay method is that the left and rightpointers can easily cross each other, which may leave the user somewhatconfused at times. Another issue is the unfavorable overall designationresolution because each touchpad covers the same screen area. In thehorizontal space, two touchpad widths (i.e. for left and right touchpad)are used to map into two screen widths.

Problems with Split Pointer Methods

As seen in FIG. 3, the basic split method typically creates a 1-to-1mapping between each touchpad and a sub-portion of the target screenarea. In effect, the right hand side pointer 32 controlled by the righthand is confined to the right hand side sub-portion 30R of the screenarea; respectively the left hand side pointer 34 controlled by the lefthand is confined to the left hand side sub-portion 30L of the screenarea. Using this method, the left and right pointer will not cross eachother, although both pointers can coexist in the middle 30M withoutcrossing. In addition, the split pointer method improves the designationresolution because two touchpads widths are mapped to one target screenarea widths. Thus, the basic split method has therefore clear advantagesover the overlay method. However, experiments have shown that usersexperience some uneasiness at times due to a perceived proximityconflict illustrated in FIG. 4.

As shown in FIG. 4, there are times when the user is naturally foundwanting to use the right pointer instead of the left pointer, for targetproximity reasons; but the fixed split imposed by the fixedsub-portioning prevents it, thus creating a mental proximity conflict.Although it would be more natural to most users to use the right pointer32 to access the input target T, the right pointer cannot be moved tothat location as it is confined to the right side portion of thedisplay.

Solution to Aforementioned Problems

The present innovation uses a novel dynamic split approach to answer theneeds mentioned above and which are:

-   -   Non-crossing left and right pointers    -   High Designation resolution    -   Capability to use any finger to point at any target coordinates

With the novel dynamic split approach, the screen addressing area forthe left and right pointers is not statically defined, but is ratheradjusted dynamically based on the actual positions of those pointers. Asshown FIGS. 5A and 5B, the addressing space R for the right hand sidepointer is bound on the left side by the left hand side pointer x-axisposition. Respectively, the addressing space L for left hand sidepointer is bound on the right side by the right hand side pointer x-axisposition. These boundaries fluctuate dynamically in real-time as thepointers are moved around.

Utilizing the dynamic split approach it is possible to implement theinvention in a variety of ways to achieve both absolute pointermanagement and relative pointer management. Several ways of doing thisare illustrated below. The basic dynamic pointer management methodillustrated in FIG. 6 continuously re-estimates the dynamic boundary sothat the right and left pointers do not cross over into the otherpointer's region. The mild pointer crossing method illustrated in FIG. 7generally enforces the dynamic approach of the absolute pointermanagement method, but permits a mild overlap of each pointer into theother pointer's region. The pointer management method illustrated inFIG. 8 enforces the basic dynamic approach, with the added feature ofimposing outer limit restrictions to improve further the effectivepointer resolution. In the detailed explanations below, the absolutepointer management case is considered first, followed by a discussion ofthe relative pointer management case.

As will be understood from the description to follow, the dynamicboundary subdivides the common display space into a left region and aright region, where the dynamic boundary defines a right edge of theleft region and a left edge of the right region. In the basic dynamicpointer management method, the dynamic boundary may be represented as agenerally vertical, one-dimensional line, such that the left and rightedges are coincident with one another (i.e., the left and right regionsabut on another). In the mild overlap dynamic pointer management method,the dynamic boundary represents a boundary region of two dimensions(e.g., such as a rectangular region) that separates the left and rightregions. In this mild overlap case, the left and right edges are spacedapart from one another, with the dynamic boundary region lying between.

Absolute Pointer Management

Basic Dynamic Boundary Method

As the result of dynamic boundaries, the remapping between touchpadcoordinates and screen coordinates must be continuously re-estimated.The novel remapping method defines a set of desired properties andconstraints to provide continuity and motion smoothness. The set belowrepresents the core constraints:

-   (1) Left_Remap(TLLx(t))==SLLx(t)-   (2) Left_Remap(TLRx(t))==SLRx(t)-   (3) SLRx(t)==SRx(t)-   (4) SRx(t)==Right_Remap(TRx(t))-   (5) Right_Remap(TRRx(t))==SRRx(t)-   (6) Right_Remap(TRLx(t))==SRLx(t)-   (7) SRLx(t)==SLx(t)-   (8) SLx(t)==Left_Remap(TLx(t))-   (9) SLRx(t)==SRx(t)-   (10) SRLx(t)==SLx(t)

The constraints (3) and (7) above create a variable stickiness whichenables reachability and continuity. It is strongest at the right handside of the left touchpad as well at the left hand side of the righttouchpad. Assuming, for instance, that the user's finger position on theright touchpad is at TRLx, any movement of the left pointer will alsomove the right pointer. The stickiness fades out when the user's fingerposition on the right touchpad goes towards TRRx.

When a normalized space is used for touchpad and screen coordinates,respectively (i.e. TLLx=0.0, TRLx=0.0, SLLx=0.0, TLRx=1.0, TRRx=1.0 andSRRx=1.0), these equalities can be reformulated as follows:

-   (1) Left_Remap(0.0)==0.0-   (2) Left_Remap(1.0)==SLRx(t)-   (3) SLRx(t)==SRx(t)-   (4) SRx(t)==Right_Remap(TRx(t))-   (5) Right_Remap(1.0)==1.0-   (6) Right_Remap(0.0)==SRLx(t)-   (7) SRLx(t)==SLx(t)-   (8) SLx(t)==Left_Remap(TLx(t))-   (9) SLRx(t)==SRx(t)-   (10) SRLx(t)==SLx(t)

An important consequence of these constraints is not to allow pointercrossing, but at the same time allow any pointer to address any desiredposition of the target screen area. A number of transformations havingthose target properties can be designed.

The preferred embodiment of this invention is defined by the followingtransformations for the left pointer and right pointer when using thenormalized space:

-   -   Left_Remap(TLx(t))=0.5*TLx(t)+0.5*TRx(t)*TLx(t)    -   Right_Remap(TRx(t))=0.5*TLx(t)+1.0*TRx(t)−0.5*TRx(t)*TLx(t)    -   which can be expressed as the following matricial product:

$\begin{bmatrix}{{SL}_{X}(t)} \\{{SR}_{X}(t)}\end{bmatrix} = {{Clip}\mspace{14mu}\left( {\begin{bmatrix}0.5 & 0 & 0.5 & 0 \\0.5 & 1 & {- 0.5} & 0\end{bmatrix} \times \begin{bmatrix}{{TL}_{X}(t)} \\{{TR}_{X}(t)} \\{{{TL}_{X}(t)}*{{TR}_{X}(t)}} \\1\end{bmatrix}} \right)}$

It corresponds to a specific quadratic systems of

$\quad\begin{bmatrix}{{TL}_{X}(t)} \\{{TR}_{X}(t)}\end{bmatrix}$where the square terms TLx(t)² and TRx(t)² are unused. The Clip( )function clips values outside the range [0.0:1.0].

The equations presented above apply to two-finger interaction but caneasily be extended to one-finger interaction. In the case where the lefthand side finger is lifted from the touchpad's surface, the equationsremain still valid by defining:

SLX(t)==0.0 t

Respectively, in the case where the right hand side finger is liftedfrom the touchpad's surface, the equations remain valid by defining:

SRX(t)==1.0 t

Other embodiments are based upon alternate constraints that are derivedby relaxing some of the core constraints.

Mild Pointer Crossing Option

An alternate embodiment with a mild pointer crossing will use thefollowing constraints:

-   (1) Left_Remap(TLLx(t))==SLLx(t)-   (2) Left_Remap(TLRx(t))==SLRx(t)-   (3) SLRx(t)==Min(1,SRx(t)+Offsetx)-   (4) SRx(t)==Right_Remap(TRx(t))-   (5) Right_Remap(TRRx(t))==SRRx(t)-   (6) Right_Remap(TRLx(t))==SRLx(t)-   (7) SRLx(t)==Max(0,SLx(t)−Offsetx)-   (8) SLx(t)==Left_Remap(TLx(t))-   (9) SLRx(t)==SRx(t)-   (10) SRLx(t)==SLx(t)    The transformation for that embodiment will then be:

$\begin{bmatrix}{{SL}_{X}(t)} \\{{SR}_{X}(t)}\end{bmatrix} = {{Clip}\mspace{14mu}\begin{pmatrix}{\begin{bmatrix}{0.5 + \alpha} & 0 & 0.5 & 0 \\0.5 & {1 + \alpha} & {- 0.5} & {- \alpha}\end{bmatrix} \times} \\\begin{bmatrix}{{TL}_{X}(t)} \\{{TR}_{X}(t)} \\{{{TL}_{X}(t)}*{{TR}_{X}(t)}} \\1\end{bmatrix}\end{pmatrix}}$where α represents the overlay control factor.

An alternate embodiment with outer limit restrictions will use thefollowing constraints:

-   (1) Left_Remap(TLLx(t))==SLLx(t)-   (2) Left_Remap(TLRx(t))==SLRx(t)-   (3) SLRx(t)==Min(SRx(t), LimitLx)-   (4) SRx(t)==Right_Remap(TRx(t))-   (5) Right_Remap(TRRx(t))==SRRx(t)-   (6) Right_Remap(TRLx(t))==SRLx(t)-   (7) SRLx(t)==Max(SLx(t), LimitRx)-   (8) SLx(t)==Left_Remap(TLx(t))-   (9) SLRx(t)==SRx(t)-   (10) SRLx(t)==SLx(t)    The transformation for that embodiment will then be:

$\begin{bmatrix}{{SL}_{X}(t)} \\{{SR}_{X}(t)}\end{bmatrix} = {{Clip}\mspace{14mu}\begin{pmatrix}{\begin{bmatrix}0.5 & 0 & {0.5 - \omega} & 0 \\{0.5 - \omega} & {1 - \omega} & {{- 0.5} + \omega} & \omega\end{bmatrix} \times} \\\begin{bmatrix}{{TL}_{X}(t)} \\{{TR}_{X}(t)} \\{{{TL}_{X}(t)}*{{TR}_{X}(t)}} \\1\end{bmatrix}\end{pmatrix}}$where ω represents the boundary control factor.Relative Pointer Management

The above descriptions focused on the absolute pointing input mode. Thefollowing description will focus on the relative mode. In relative mode,the same non-crossing pointer constraints (i.e. dynamic boundarymanagement) are preserved but touchpad finger motion is interpreted asan offset with respect to the current screen pointer position. Theoffset amount is in the preferred embodiment proportional to the widthof the current screen area, that is to say, proportional to:

WLx(t)=SLRx(t)−SLLx(t) for the left pointer

and to:

WRx(t)=SRRx(t)−SRLx(t) for the right pointer

The effect of this proportionality is 1) to enable a better designationprecision when the screen area is smaller and 2) to enable a fasteraccess when the screen area is larger. FIG. 9 shows the effect of a sameswipe motion on the left and right touchpad which results in differentoffsets in the screen space. In relative mode, the displacement offsetin the screen space illustrated is twice as important for the rightpointer as for the left pointer for the same swipe motion on therespective touchpad since the width of right interactive area is twiceas wide as the left interactive area.

Implementation Examples

The pointer management techniques described above can be implemented ina variety of different physical embodiments. By way of example, refer toFIG. 10, which illustrates how the remote control device of FIG. 1A maybe implemented. As illustrated, touchpad A and touchpad B correspond tothe clickable touchpads 20, 22 of FIG. 1A. These may be implemented ascapacitive touch surfaces that are each mounted in a detent structurethat allows the respective touch surface to be clicked by a pushingforce applied normal to the touchpad surface. Such clicking motionactivates a micro-switch mounted beneath the touch pad. The micro-switchprovides a momentary binary signal each time the pad is clicked.

In the multi-touch embodiment of FIG. 1B, the multi-touch surface isable to individually detect and discriminate between contact from theright and left hands and thus provides a separate (x,y) position datapoint for each finger in contact with the touchpad surface. Userselection of a desired (x,y) position may be effected by sensing amomentary tap by the user's finger, for example.

In the illustrated embodiment of FIG. 10, the touchpads A and B eachprovide a first output signal indicative of the (x,y) position wherefinger contact is made and a second output signal that carries themicro-switch click data. These outputs are fed to the dual port touchpadinterface 50, which provides a digital interface to receive each of theoutput signals from each of the touchpads. The interface 50 associatesidentification or address data to the received signals, to distinguishsignals of touchpad A (i.e., left hand touchpad 20 of FIG. 1A) fromsignals of touchpad B (i.e., right hand touchpad 22 of FIG. 1A). Theaddress-associated data are fed from interface 50 to a serial input ofthe processor (CPU) 52. Processor (CPU) 52 may be implemented as amicrocontroller, microprocessor, or the like. Coupled to the processor52 is a first memory (e.g., EEPROM memory) 54 that stores the operatinginstructions that control how the processor functions. A flowchartdiagram illustrating these instructions is shown in FIG. 11. To provideworking memory to perform the matrix transformations described above, asuitable second memory (e.g., RAM memory) 56 is provided. In FIG. 10,the transformation matrix has been diagrammatically illustrated at 57.

Also coupled to the processor is the button array 58. The button arrayrepresents the set of application-definable buttons (buttons 24 of FIG.1A). The button array may be implemented using suitable micro-switches,membrane switches or the like. The button array may be coupled to theprocessor 52 via the serial interface, as shown, or the individualbuttons of the array may each be coupled via dedicated digital inputs ofthe processor (not shown).

To effect control over the pointer display system, the remote controldevice of FIG. 10 includes a wireless transmitter 60 which communicateswith a corresponding wireless receiver 62. Wireless communication can beeffected using radio frequency (RF) signals, infrared (IR) signals, orboth. Based on the matrix transformations described above, the processorcomputes the respective positions for each of the pointers, using themapping integrator functionality to compute the positions so that adynamic boundary between the pointer is adjusted based on knowledge ofthe respective (x,y) positions of the touchpad output signals. Theprocessor transmits these pointer position data via the transmitter 60and receiver 62 to the graphics generator circuitry 64. The graphicsgenerator circuitry is designed to generate graphical images of therespective pointers for display within the display space or displayregion 66 of a display device 68. The graphics generator thus suppliesthe bit-mapped or vector graphics data to render the image of thepointer. The point on the screen at which that image is rendered isbased on information received wirelessly from the remote control device.Of course, if desired, a hardwired connection may be used between theprocessor 52 and the graphics generator 64. In such embodiment, thewireless transmitter and wireless receiver would not be used.

As an alternative to the processor-based embodiment described above, thecontroller circuitry described above can be implemented using anapplication specific integrated circuit (ASIC), if desired.

In the embodiments depicted above, the mapping integrator functionalityis provided by the processor within the remote control device. However,if desired, the mapping integrator functionality can alternatively beplaced in the controlled, consumer electronic device (e.g., set top box,television, computer system). In such an alternate embodiment, theposition signals are transmitted to the controlled device and anon-board processor or ASIC associated with the controlled deviceperforms the mapping integrator functionality.

FIG. 11 provides a flow diagram illustrating how the processor 52 isprogrammed and how the graphics generator 64 handles the instructionsreceived from the processor. In step 100, the processor receives inputfrom the input devices (in this case, touchpads A and B). These inputvalues give touchpad coordinates (x,y) and have corresponding addressesdenoting which touchpad supplied the data.

In step 102, the processor processes these input data by constructing atransformation matrix 57 in RAM based on the transformation matrix modelbeing used. The written description above has explained in detailvarious different transformation matrices that may be used based on thedesired behavior of the system (absolute-relative; basic, mild pointercrossing enabled, with/without outer limit constraints). It will beunderstood that the processor computes the pointer positions byperforming matrix multiplication of the input data, based on the matrixused.

Thus, in step 104 the processor computes the pointer positions of therespective pointer positions based on the transformed inputs that wereprocessed by the transformation matrix. These pointer positions are thenmapped into the display space of the display screen using the commoncoordinate system of the display space.

In step 106, the pointer positions are then transmitted to the graphicsgenerator. The graphics generator 64 then performs the following steps:

In step 108, graphical images are generated for each pointer to bedisplayed. As stated above, these graphical images can be generatedusing bitmapped graphics, where the pointer images are pre-stored inmemory, or by using vector graphic techniques to generate the images onthe fly. In either case, the images are placed on the display screen atlocations within the common display space of the screen based on thelocations specified in step 104, as depicted at step 110.

Functional Analysis of Dynamic Controller Methods

From a functional perspective, the dual pointer (or multi-pointer)management approach uses data from both (all) input devices to computethe respective positions of both (all) of the pointers. In effect, theinput data are cross-coupled via the transformation matrix, so that thecomputed pointer positions are interdependent. In this way, the systemgenerates a dynamic boundary within the common display coordinatesystem, where the dynamic boundary is adjusted by the processor based onknowledge of the respective position signals from the input devices.

To illustrate, see FIG. 12, which presents a functional block diagram ofa pointer control apparatus in accordance with the disclosedinnovations. Note that the input devices (device #1 and device #2) andtheir associated controllers each supply a position signal, on signallines 200 and 202, respectively. These are fed to the mapping integratorthat generates pointer position data for the left pointer and rightpointer on data lines 206 and 208, respectively. These pointer positiondata are then used by the left pointer controller 210 and right pointercontroller 212 to generate the respective left and right pointers fordisplay within the display space 66. The mapping integrator computes theposition of the first pointer within the display coordinate system basedin part upon the position signal of the first input device and based inpart upon the position signal of the second input device, and computesthe position of the second pointer within the display coordinate systembased in part upon the position signal of the second input device andbased in part upon the position signal of the first input device.

By way of further illustration, refer to FIG. 13, which shows amulti-device system. In this example, three input devices areillustrated, but it will be understood that the multi-device system canhave any plural number of input devices (i.e, 2 or more input devices).In this embodiment, a single multi-position input system is employed.This system receives input position signals from each of the inputdevices 222 and supplies data representing these input signals to themapping integrator 204. The multi-position input system receives each ofthe input device signals and appends address information to each so thatthe data take the form of (addr, x, y). These data are then sent in aserial stream to the mapping integrator 204. The mapping integratorfunctions as described above to generate the respective pointer positionsignals. In addition, the mapping integrator also appends pointeridentification tags to each of the pointer positions calculated. Theoutput of the mapping integrator thus takes the form of (pointer tag, x,y) where the pointer tag indicates to which pointer the data corresponds(left, right, middle, etc.) and the x,y values represent positionswithin the common display coordinate system (i.e., within the displayspace). The pointer generation system 205 receives the output data ofthe mapping integrator, parses the data by pointer tag and thengenerates and displays the pointer images within the display space.

Conclusion

From the foregoing, it will be appreciated that the present innovationemploys novel dynamic boundary control techniques to solve many problemsarising in multi-pointer control applications. The innovation enables amore intuitive interaction model for dual pointer-based input which isbased on a non-crossing (or mildly crossing) pointer approach and thatallows users to use any pointer to reach any area of the display. Theinput devices can be any pointing device that provides at least onedegree or more of movement, such as touchpads, touch screens andjoysticks. The innovation also provides input speed advantages whencompared to traditional input methods due its dynamicprecision/flexibility tradeoff management.

Typical applications for dual pointer input are text entry using avirtual keyboard displayed on screen, or multiple object selectiontasks, for instance. The dual pointer techniques are also well suited togaming applications as well.

The description of the invention is merely exemplary in nature and,thus, variations that do not depart from the gist of the invention areintended to be within the scope of the invention. Such variations arenot to be regarded as a departure from the spirit and scope of theinvention. For example, while the matricial transformation (using atransformation matrix) has been utilized in some of the embodimentsdescribed herein, other ways of performing these transformations mayalternatively be employed.

1. A pointer control apparatus comprising: a first input device thatprovides a position signal expressed using a first coordinate systemassociated with the first input device; a second input device thatprovides a position signal expressed using a second coordinate systemassociated with the second input device; a pointer generation systemthat generates for display in a common display space having a commondisplay coordinate system a first pointer associated with the firstinput device and a second pointer associated with the second inputdevice; a mapping integrator processor configured to compute theposition of the first pointer within the display coordinate system basedin part upon the position signal of the first input device and based inpart upon the position signal of the second input device, and configuredto compute the position of the second pointer within the displaycoordinate system based in part upon the position signal of the secondinput device and based in part upon the position signal of the firstinput device; the mapping integrator processor being further configuredto compute the position of the first and second pointers such that eachis confined to a different side of a dynamic boundary within the commondisplay coordinate system, where the dynamic boundary is adjusted by theprocessor based on knowledge of the respective position signals of thefirst and second input devices.
 2. The pointer control apparatus ofclaim 1 wherein the dynamic boundary subdivides the common display spaceinto two non-overlapping contiguous regions where the union of the tworegions equals the common display space.
 3. The pointer controlapparatus of claim 1 wherein the dynamic boundary subdivides the commondisplay space into two partially overlapping contiguous regions.
 4. Thepointer control apparatus of claim 1 wherein the dynamic boundarysubdivides the common display space into two contiguous regions that arestatically constrained along their dynamic extremities.
 5. The pointercontrol apparatus of claim 1 wherein the first input device provides aleft position signal and the second input device provides a rightposition signal; wherein the dynamic boundary subdivides the commondisplay space into a left region and a right region, said dynamicboundary defining a right edge of said left region and a left edge ofsaid right region; and wherein the mapping integrator processor computessaid right edge based on the right position signal and computes saidleft edge based on the left position signal.
 6. The pointer controlapparatus of claim 5 wherein said left and right edges are coincidentwith one another.
 7. The pointer control apparatus of claim 5 whereinsaid left and right edges are spaced apart from on another.
 8. Thepointer control apparatus of claim 1 wherein said first and second inputdevices operate independently of one another.
 9. The pointer controlapparatus of claim 1 wherein said first and second input devicesgenerate position signals using sensors of the same type.
 10. Thepointer control apparatus of claim 1 wherein said first and second inputdevices generate position signals using sensors of different types. 11.The pointer control apparatus of claim 1 further comprising at least oneadditional input device that provides an additional input signal;wherein said pointer generation system generates a pointer associatedwith said additional input device; and wherein said mapping integratorprocessor computes the position of the pointer associated with saidadditional input device based on the position signals of the first andsecond input devices as well as the additional input signal.
 12. Thepointer control apparatus of claim 1 wherein at least one of said firstand second coordinate systems is a one-dimensional coordinate system.13. The pointer control apparatus of claim 1 wherein at least one ofsaid first and second coordinate systems is a two-dimensional coordinatesystem.
 14. The pointer control apparatus of claim 1 wherein at leastone of said first and second coordinate systems is a three-dimensionalcoordinate system.
 15. The pointer control apparatus of claim 1 whereinsaid mapping integrator processor performs at least one transformation.16. The pointer control apparatus of claim 4 wherein said mappingintegrator processor performs at least one transformation that includesan overlay control factor that permits both first and second pointers tosimultaneously occupy said intermediate region.
 17. The pointer controlapparatus of claim 1 wherein said mapping integrator processor performsat least one transformation that includes an outer limit restrictionwhereby predefined positions of the first and second input devices arenot mapped to the display coordinate system.
 18. The pointer controlapparatus of claim 1 wherein said processor is selected from the groupconsisting of microprocessor, microcontroller, digital signal processor,computer, and application specific integrated circuit (ASIC).
 19. Thepointer control apparatus of claim 1 wherein said processor is embeddedin at least one of said first and second input devices.
 20. The pointercontrol apparatus of claim 1 wherein said processor is configured byprogram instructions stored in a machine-readable memory coupled to saidprocessor.
 21. A method of generating plural pointers on a displayscreen that are responsive to plural input devices, comprising:electronically receiving a first position signal from a first inputdevice; electronically receiving a second position signal from a secondinput device; electronically processing said first and second positionsignals to calculate first and second pointer positions within apredefined display coordinate system associated with a display screen,the first and second pointer positions being different from one anotherand each being based on both said first and second position signals;generating and displaying first and second pointers on said displayscreen, the first pointer being displayed at said first pointer positionand the second pointer being displayed at said second pointer position;wherein said processing step is performed such that the first and secondpointer positions are confined to different sides of a dynamic boundarywithin the display coordinate system, wherein the dynamic boundary iscomputationally adjusted based on the first and second position signals.22. The method of claim 21 wherein said first position signal is a leftposition signal and the second position signal is a right positionsignal; wherein said processing step is performed such that said dynamicboundary subdivides a display space into a left region and a rightregion, said dynamic boundary defining a right edge of said left regionand a left edge of said right region; and wherein said processing stepfurther includes computing said right edge based on the right positionsignal and computing said left edge based on the left position signal.23. The method of claim 21 wherein said processing step is performed byapplying a transformation of said first and second position signals. 24.The method of claim 21 wherein said processing step is performed byapplying a transformation that defines a quadratic relationship amongthe first and second position signals.
 25. The method of claim 21wherein said processing step is performed by applying a transformationthat defines a quadratic relationship among the first and secondposition signals and that includes an overlay control factor thatpermits said first and second pointer positions to occupy a commonintermediate region associated with said dynamic boundary.
 26. Themethod of claim 21 wherein said first and second input devices each havean associated range of selectable positions; and wherein said processingstep is performed by applying a transformation that defines a quadraticrelationship among the first and second position signals and thatincludes an outer limit restriction whereby predefined portions of theassociated ranges of said first and second input devices are not mappedto the display coordinate system.
 27. The method of claim 21 whereinsaid processing step is performed to define a dynamic linear boundarythat subdivides the display screen into two contiguous regions.
 28. Themethod of claim 21 wherein said processing step is performed to define adynamic curvilinear boundary that subdivides the display screen into twocontiguous regions.
 29. The method of claim 21 wherein said processingstep is performed to define a dynamic boundary that comprises anintermediate region that subdivides the display screen into twocontiguous regions on opposite sides of the intermediate region.
 30. Themethod of claim 21 wherein said electronically processing is performedusing a device selected from the group consisting of microprocessor,microcontroller, digital signal processor, computer, and applicationspecific integrated circuit (ASIC).
 31. An apparatus for generatingplural pointers on a display screen comprising: a multi-position inputsystem receptive of position input data from a plurality of inputdevices; a pointer generation system that generates plural pointers fordisplay on said display screen each at a different pointer location; amapping integrator processor configured to compute said differentpointer locations by taking into account the position input data fromall of said plurality of input devices in computing each pointerlocation; the mapping integrator processor being further configured tocompute said different pointer locations such that each location lies ona different side of a dynamic boundary that is adjusted by the processorbased on knowledge of the input data from said plurality of inputdevices.